{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pipeline example with OpenVINO inference execution engine \n",
    "\n",
    "This notebook illustrates how you can serve ensemble of models using [OpenVINO prediction model](https://github.com/SeldonIO/seldon-core/tree/master/wrappers/s2i/python_openvino).\n",
    "The demo includes optimized ResNet50 and DenseNet169 models by OpenVINO model optimizer. \n",
    "They have [reduced precision](https://www.intel.ai/introducing-int8-quantization-for-fast-cpu-inference-using-openvino/#gs.lUSgiWKa) of graph operations from FP32 to INT8. It significantly improves the execution peformance with minimal impact on the accuracy. The gain is particulary visible with the latest Casade Lake CPU with [VNNI](https://www.intel.ai/intel-deep-learning-boost/#gs.sy7JEtwu) extension.\n",
    "\n",
    "![pipeline](pipeline1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Seldon Core on Minikube or on any Kubernetes cluster\n",
    "\n",
    "**The Minikube example below assumes version 0.30.0 installed**\n",
    "\n",
    "It also assumes;\n",
    "  * You have 4G of memory available\n",
    "  * You have 4 CPU Cores available\n",
    "  * You have 20G of free disk\n",
    "  \n",
    "**If you already have Kubernetes cluster present, you can skip minikube setup steps**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting local Kubernetes v1.13.2 cluster...\n",
      "Starting VM...\n",
      "Getting VM IP address...\n",
      "Moving files into cluster...\n",
      "Setting up certs...\n",
      "Connecting to cluster...\n",
      "Setting up kubeconfig...\n",
      "Stopping extra container runtimes...\n",
      "Starting cluster components...\n",
      "Verifying kubelet health ...\n",
      "Verifying apiserver health ...\n",
      "Kubectl is now configured to use the cluster.\n",
      "Loading cached images from config file.\n",
      "\n",
      "\n",
      "Everything looks great. Please enjoy minikube!\n"
     ]
    }
   ],
   "source": [
    "!minikube start --memory 4096 --cpus 4 --disk-size 20g"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "namespace/seldon created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create namespace seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context \"minikube\" modified.\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl config set-context $(kubectl config current-context) --namespace=seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$HELM_HOME has been configured at /home/clive/.helm.\n",
      "\n",
      "Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.\n",
      "\n",
      "Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.\n",
      "To prevent this, run `helm init` with the --tiller-tls-verify flag.\n",
      "For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation\n",
      "Happy Helming!\n"
     ]
    }
   ],
   "source": [
    "!helm init"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"tiller-deploy\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"tiller-deploy\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/tiller-deploy -n kube-system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   seldon-core\n",
      "LAST DEPLOYED: Wed Apr 24 15:19:35 2019\n",
      "NAMESPACE: seldon-system\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1beta1/CustomResourceDefinition\n",
      "NAME                                         AGE\n",
      "seldondeployments.machinelearning.seldon.io  1s\n",
      "\n",
      "==> v1/ClusterRole\n",
      "seldon-operator-manager-role  1s\n",
      "\n",
      "==> v1/ClusterRoleBinding\n",
      "NAME                                 AGE\n",
      "seldon-operator-manager-rolebinding  1s\n",
      "\n",
      "==> v1/Service\n",
      "NAME                                        TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)  AGE\n",
      "seldon-operator-controller-manager-service  ClusterIP  10.107.189.217  <none>       443/TCP  1s\n",
      "\n",
      "==> v1/StatefulSet\n",
      "NAME                                DESIRED  CURRENT  AGE\n",
      "seldon-operator-controller-manager  1        1        1s\n",
      "\n",
      "==> v1/Pod(related)\n",
      "NAME                                  READY  STATUS             RESTARTS  AGE\n",
      "seldon-operator-controller-manager-0  0/1    ContainerCreating  0         1s\n",
      "\n",
      "==> v1/Secret\n",
      "NAME                                   TYPE    DATA  AGE\n",
      "seldon-operator-webhook-server-secret  Opaque  0     1s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "NOTES: TODO\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../../../helm-charts/seldon-core-operator --name seldon-core --set usageMetrics.enabled=true   --namespace seldon-system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "partitioned roll out complete: 1 new pods have been updated...\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/seldon-controller-manager -n seldon-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Ingress\n",
    "Please note: There are reported gRPC issues with ambassador (see https://github.com/SeldonIO/seldon-core/issues/473)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   ambassador\n",
      "LAST DEPLOYED: Wed Apr 24 15:20:31 2019\n",
      "NAMESPACE: seldon\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1/Service\n",
      "NAME               TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE\n",
      "ambassador-admins  ClusterIP     10.110.65.117  <none>       8877/TCP                    0s\n",
      "ambassador         LoadBalancer  10.96.204.128  <pending>    80:32059/TCP,443:31261/TCP  0s\n",
      "\n",
      "==> v1/Deployment\n",
      "NAME        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE\n",
      "ambassador  3        3        3           0          0s\n",
      "\n",
      "==> v1/Pod(related)\n",
      "NAME                         READY  STATUS             RESTARTS  AGE\n",
      "ambassador-5b89d44544-46tmd  0/1    ContainerCreating  0         0s\n",
      "ambassador-5b89d44544-dlvhg  0/1    ContainerCreating  0         0s\n",
      "ambassador-5b89d44544-nhzj2  0/1    ContainerCreating  0         0s\n",
      "\n",
      "==> v1/ServiceAccount\n",
      "NAME        SECRETS  AGE\n",
      "ambassador  1        0s\n",
      "\n",
      "==> v1beta1/ClusterRole\n",
      "NAME        AGE\n",
      "ambassador  0s\n",
      "\n",
      "==> v1beta1/ClusterRoleBinding\n",
      "NAME        AGE\n",
      "ambassador  0s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "Congratuations! You've successfully installed Ambassador.\n",
      "\n",
      "For help, visit our Slack at https://d6e.co/slack or view the documentation online at https://www.getambassador.io.\n",
      "\n",
      "To get the IP address of Ambassador, run the following commands:\n",
      "NOTE: It may take a few minutes for the LoadBalancer IP to be available.\n",
      "     You can watch the status of by running 'kubectl get svc -w  --namespace seldon ambassador'\n",
      "\n",
      "  On GKE/Azure:\n",
      "  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].ip}')\n",
      "\n",
      "  On AWS:\n",
      "  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')\n",
      "\n",
      "  echo http://$SERVICE_IP:\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install stable/ambassador --name ambassador --set crds.keep=false"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"ambassador\" rollout to finish: 0 of 3 updated replicas are available...\n",
      "Waiting for deployment \"ambassador\" rollout to finish: 1 of 3 updated replicas are available...\n",
      "Waiting for deployment \"ambassador\" rollout to finish: 2 of 3 updated replicas are available...\n",
      "deployment \"ambassador\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deployment.apps/ambassador"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Optional) Install Jaeger\n",
    "\n",
    "We will use the Jaeger All-in-1 resource found at the [Jaeger Kubernetes repo](https://github.com/jaegertracing/jaeger-kubernetes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deployment.extensions/jaeger created\n",
      "service/jaeger-query created\n",
      "service/jaeger-collector created\n",
      "service/jaeger-agent created\n",
      "service/zipkin created\n"
     ]
    }
   ],
   "source": [
    "!kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start Jaeger UI\n",
    "\n",
    "```\n",
    "minikube service jaeger-query -n seldon\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Optional) Build Model, Combiner and Transformer Images\n",
    "This is optional step. You can skip building the docker images for the pipeline components and rely on the prebuilt versions in the public docker registry. \n",
    "\n",
    "The commands below build the components on the docker registry inside the minikube.\n",
    "\n",
    "Alternatively you can change the REGISTRY variable to your private one and drop the `eval $(minikube docker-env) &&` phrase. In that case, after the images are built, you need to push them to your docker registry and update the images names in the pipeline json file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: REGISTRY=docker.io/seldonio\n"
     ]
    }
   ],
   "source": [
    "%env REGISTRY=docker.io/seldonio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---> Installing application source...\n",
      "---> Installing dependencies ...\n",
      "Looking in links: /whl\n",
      "Collecting google-cloud-storage==1.13.0 (from -r requirements.txt (line 1))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/d7/62/a2e3111bf4d1eb54fe86dec694418644e024eb059bf1e66ebdcf9f98ad70/google_cloud_storage-1.13.0-py2.py3-none-any.whl (59kB)\n",
      "Collecting boto3==1.9.34 (from -r requirements.txt (line 2))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/94/04/c48c102e11b0cb2e3d4a7bdda49647b40e2ae03279ce9ba935e4ae66ab89/boto3-1.9.34-py2.py3-none-any.whl (128kB)\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Collecting google-cloud-core<0.29dev,>=0.28.0 (from google-cloud-storage==1.13.0->-r requirements.txt (line 1))\n",
      "Downloading https://files.pythonhosted.org/packages/0f/41/ae2418b4003a14cf21c1c46d61d1b044bf02cf0f8f91598af572b9216515/google_cloud_core-0.28.1-py2.py3-none-any.whl\n",
      "Collecting google-api-core<2.0.0dev,>=0.1.1 (from google-cloud-storage==1.13.0->-r requirements.txt (line 1))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/8b/01/13758ff9b970008ccf9e0dcc3b86d0e01937d7485b9a2c6142c9c2bdb4da/google_api_core-1.7.0-py2.py3-none-any.whl (64kB)\n",
      "Collecting google-resumable-media>=0.3.1 (from google-cloud-storage==1.13.0->-r requirements.txt (line 1))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/e2/5d/4bc5c28c252a62efe69ed1a1561da92bd5af8eca0cdcdf8e60354fae9b29/google_resumable_media-0.3.2-py2.py3-none-any.whl\n",
      "Collecting botocore<1.13.0,>=1.12.34 (from boto3==1.9.34->-r requirements.txt (line 2))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/c8/6c/2058039815eb4eac4f2f7462ecae3e352e994d6618ba1f27114d9b985618/botocore-1.12.89-py2.py3-none-any.whl (5.2MB)\n",
      "Collecting jmespath<1.0.0,>=0.7.1 (from boto3==1.9.34->-r requirements.txt (line 2))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/b7/31/05c8d001f7f87f0f07289a5fc0fc3832e9a57f2dbd4d3b0fee70e0d51365/jmespath-0.9.3-py2.py3-none-any.whl\n",
      "Collecting s3transfer<0.2.0,>=0.1.10 (from boto3==1.9.34->-r requirements.txt (line 2))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/d7/14/2a0004d487464d120c9fb85313a75cd3d71a7506955be458eebfe19a6b1d/s3transfer-0.1.13-py2.py3-none-any.whl (59kB)\n",
      "Requirement already satisfied: setuptools>=34.0.0 in /opt/conda/lib/python3.6/site-packages (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage==1.13.0->-r requirements.txt (line 1)) (40.8.0)\n",
      "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.6/site-packages (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage==1.13.0->-r requirements.txt (line 1)) (2.18.4)\n",
      "Requirement already satisfied: protobuf>=3.4.0 in /opt/conda/lib/python3.6/site-packages (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage==1.13.0->-r requirements.txt (line 1)) (3.6.1)\n",
      "Collecting googleapis-common-protos!=1.5.4,<2.0dev,>=1.5.3 (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage==1.13.0->-r requirements.txt (line 1))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/61/29/1549f61917eadd11650e42b78b4afcfe9cb467157af4510ab8cb59535f14/googleapis-common-protos-1.5.6.tar.gz\n",
      "Requirement already satisfied: six>=1.10.0 in /opt/conda/lib/python3.6/site-packages (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage==1.13.0->-r requirements.txt (line 1)) (1.11.0)\n",
      "Collecting pytz (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage==1.13.0->-r requirements.txt (line 1))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl (510kB)\n",
      "Collecting google-auth<2.0dev,>=0.4.0 (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage==1.13.0->-r requirements.txt (line 1))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/4e/85/71b2dfbf5b4241cd031cc333ed71f90a271074a97cb2c517bb65f07a1a90/google_auth-1.6.2-py2.py3-none-any.whl (73kB)\n",
      "Collecting docutils>=0.10 (from botocore<1.13.0,>=1.12.34->boto3==1.9.34->-r requirements.txt (line 2))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/36/fa/08e9e6e0e3cbd1d362c3bbee8d01d0aedb2155c4ac112b19ef3cae8eed8d/docutils-0.14-py3-none-any.whl (543kB)\n",
      "Collecting python-dateutil<3.0.0,>=2.1; python_version >= \"2.7\" (from botocore<1.13.0,>=1.12.34->boto3==1.9.34->-r requirements.txt (line 2))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)\n",
      "Requirement already satisfied: urllib3<1.25,>=1.20; python_version >= \"3.4\" in /opt/conda/lib/python3.6/site-packages (from botocore<1.13.0,>=1.12.34->boto3==1.9.34->-r requirements.txt (line 2)) (1.22)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage==1.13.0->-r requirements.txt (line 1)) (3.0.4)\n",
      "Requirement already satisfied: idna<2.7,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage==1.13.0->-r requirements.txt (line 1)) (2.6)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage==1.13.0->-r requirements.txt (line 1)) (2018.1.18)\n",
      "Collecting cachetools>=2.0.0 (from google-auth<2.0dev,>=0.4.0->google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage==1.13.0->-r requirements.txt (line 1))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/39/2b/d87fc2369242bd743883232c463f28205902b8579cb68dcf5b11eee1652f/cachetools-3.1.0-py2.py3-none-any.whl\n",
      "Collecting pyasn1-modules>=0.2.1 (from google-auth<2.0dev,>=0.4.0->google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage==1.13.0->-r requirements.txt (line 1))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/da/98/8ddd9fa4d84065926832bcf2255a2b69f1d03330aa4d1c49cc7317ac888e/pyasn1_modules-0.2.4-py2.py3-none-any.whl (66kB)\n",
      "Collecting rsa>=3.1.4 (from google-auth<2.0dev,>=0.4.0->google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage==1.13.0->-r requirements.txt (line 1))\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Downloading https://files.pythonhosted.org/packages/02/e5/38518af393f7c214357079ce67a317307936896e961e35450b70fad2a9cf/rsa-4.0-py2.py3-none-any.whl\n",
      "  Url '/whl' is ignored. It is either a non-existing path or lacks a specific scheme.\n",
      "Collecting pyasn1<0.5.0,>=0.4.1 (from pyasn1-modules>=0.2.1->google-auth<2.0dev,>=0.4.0->google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage==1.13.0->-r requirements.txt (line 1))\n",
      "Downloading https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl (73kB)\n",
      "Building wheels for collected packages: googleapis-common-protos\n",
      "Running setup.py bdist_wheel for googleapis-common-protos: started\n",
      "Running setup.py bdist_wheel for googleapis-common-protos: finished with status 'done'\n",
      "Stored in directory: /root/.cache/pip/wheels/da/6b/81/8573adcbe2aa2ecba92c341dfe19c5b5a733f4514297ba52b4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully built googleapis-common-protos\n",
      "Installing collected packages: googleapis-common-protos, pytz, cachetools, pyasn1, pyasn1-modules, rsa, google-auth, google-api-core, google-cloud-core, google-resumable-media, google-cloud-storage, jmespath, docutils, python-dateutil, botocore, s3transfer, boto3\n",
      "Successfully installed boto3-1.9.34 botocore-1.12.89 cachetools-3.1.0 docutils-0.14 google-api-core-1.7.0 google-auth-1.6.2 google-cloud-core-0.28.1 google-cloud-storage-1.13.0 google-resumable-media-0.3.2 googleapis-common-protos-1.5.6 jmespath-0.9.3 pyasn1-0.4.5 pyasn1-modules-0.2.4 python-dateutil-2.8.0 pytz-2018.9 rsa-4.0 s3transfer-0.1.13\n",
      "You are using pip version 10.0.1, however version 19.0.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n",
      "Build completed successfully\n"
     ]
    }
   ],
   "source": [
    "!eval $(minikube docker-env) && cd resources/model && s2i build -E environment_grpc . ${REGISTRY}/seldon-core-s2i-openvino:0.1 ${REGISTRY}/seldon-openvino-prediction:0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---> Installing application source...\n",
      "Build completed successfully\n"
     ]
    }
   ],
   "source": [
    "!eval $(minikube docker-env) && cd resources/combiner && s2i build -E environment_grpc . ${REGISTRY}/seldon-core-s2i-openvino:0.1 ${REGISTRY}/imagenet_combiner:0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---> Installing application source...\n",
      "---> Installing dependencies ...\n",
      "Looking in links: /whl\n",
      "You are using pip version 10.0.1, however version 19.0.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n",
      "Build completed successfully\n"
     ]
    }
   ],
   "source": [
    "!eval $(minikube docker-env) && cd resources/transformer && s2i build -E environment_grpc . ${REGISTRY}/seldon-core-s2i-openvino:0.1 ${REGISTRY}/imagenet_transformer:0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy Seldon pipeline with Intel OpenVINO models ensemble\n",
    "\n",
    " * Ingest compressed JPEG binary and transform to TensorFlow Proto payload\n",
    " * Ensemble two OpenVINO optimized models for ImageNet classification: ResNet50, DenseNet169\n",
    " * Return result in human readable text\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: graphviz in /home/clive/anaconda3/lib/python3.6/site-packages (0.8.2)\n",
      "\u001b[33mYou are using pip version 19.0.3, however version 19.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install graphviz\n",
    "import sys\n",
    "sys.path.append(\"../../../notebooks\")\n",
    "from visualizer import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"300pt\" height=\"387pt\"\n",
       " viewBox=\"0.00 0.00 300.00 387.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 383)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-383 296,-383 296,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\"><title>cluster_0</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"8,-8 8,-371 284,-371 284,-8 8,-8\"/>\n",
       "<text text-anchor=\"middle\" x=\"146\" y=\"-355.8\" font-family=\"Times,serif\" font-size=\"14.00\">predictor&#45;0</text>\n",
       "</g>\n",
       "<!-- imagenet&#45;otransformer0 -->\n",
       "<g id=\"node1\" class=\"node\"><title>imagenet&#45;otransformer0</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"200,-340 58,-340 58,-304 200,-304 200,-340\"/>\n",
       "<text text-anchor=\"middle\" x=\"129\" y=\"-318.3\" font-family=\"Times,serif\" font-size=\"14.00\">imagenet&#45;otransformer</text>\n",
       "</g>\n",
       "<!-- imagenet&#45;otransformer0endpoint -->\n",
       "<g id=\"node2\" class=\"node\"><title>imagenet&#45;otransformer0endpoint</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"50\" cy=\"-250\" rx=\"33.5952\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"50\" y=\"-246.3\" font-family=\"Times,serif\" font-size=\"14.00\">GRPC</text>\n",
       "</g>\n",
       "<!-- imagenet&#45;otransformer0&#45;&gt;imagenet&#45;otransformer0endpoint -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>imagenet&#45;otransformer0&#45;&gt;imagenet&#45;otransformer0endpoint</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M109.472,-303.697C98.8828,-294.314 85.6952,-282.629 74.4308,-272.648\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"76.439,-269.751 66.6332,-265.738 71.7966,-274.99 76.439,-269.751\"/>\n",
       "</g>\n",
       "<!-- imagenet&#45;itransformer -->\n",
       "<g id=\"node3\" class=\"node\"><title>imagenet&#45;itransformer</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"240.5,-268 101.5,-268 101.5,-232 240.5,-232 240.5,-268\"/>\n",
       "<text text-anchor=\"middle\" x=\"171\" y=\"-246.3\" font-family=\"Times,serif\" font-size=\"14.00\">imagenet&#45;itransformer</text>\n",
       "</g>\n",
       "<!-- imagenet&#45;otransformer0&#45;&gt;imagenet&#45;itransformer -->\n",
       "<g id=\"edge9\" class=\"edge\"><title>imagenet&#45;otransformer0&#45;&gt;imagenet&#45;itransformer</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M139.382,-303.697C144.265,-295.559 150.187,-285.689 155.579,-276.701\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"158.594,-278.48 160.737,-268.104 152.591,-274.879 158.594,-278.48\"/>\n",
       "</g>\n",
       "<!-- imagenet&#45;itransformerendpoint -->\n",
       "<g id=\"node4\" class=\"node\"><title>imagenet&#45;itransformerendpoint</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"58\" cy=\"-178\" rx=\"33.5952\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"58\" y=\"-174.3\" font-family=\"Times,serif\" font-size=\"14.00\">GRPC</text>\n",
       "</g>\n",
       "<!-- imagenet&#45;itransformer&#45;&gt;imagenet&#45;itransformerendpoint -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>imagenet&#45;itransformer&#45;&gt;imagenet&#45;itransformerendpoint</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M143.357,-231.876C126.554,-221.467 105.096,-208.174 87.8981,-197.521\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"89.6848,-194.511 79.3405,-192.22 85.9984,-200.461 89.6848,-194.511\"/>\n",
       "</g>\n",
       "<!-- imagenet&#45;combiner -->\n",
       "<g id=\"node5\" class=\"node\"><title>imagenet&#45;combiner</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"232.5,-196 109.5,-196 109.5,-160 232.5,-160 232.5,-196\"/>\n",
       "<text text-anchor=\"middle\" x=\"171\" y=\"-174.3\" font-family=\"Times,serif\" font-size=\"14.00\">imagenet&#45;combiner</text>\n",
       "</g>\n",
       "<!-- imagenet&#45;itransformer&#45;&gt;imagenet&#45;combiner -->\n",
       "<g id=\"edge8\" class=\"edge\"><title>imagenet&#45;itransformer&#45;&gt;imagenet&#45;combiner</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M171,-231.697C171,-223.983 171,-214.712 171,-206.112\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"174.5,-206.104 171,-196.104 167.5,-206.104 174.5,-206.104\"/>\n",
       "</g>\n",
       "<!-- imagenet&#45;combinerendpoint -->\n",
       "<g id=\"node6\" class=\"node\"><title>imagenet&#45;combinerendpoint</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"50\" cy=\"-106\" rx=\"33.5952\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"50\" y=\"-102.3\" font-family=\"Times,serif\" font-size=\"14.00\">GRPC</text>\n",
       "</g>\n",
       "<!-- imagenet&#45;combiner&#45;&gt;imagenet&#45;combinerendpoint -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>imagenet&#45;combiner&#45;&gt;imagenet&#45;combinerendpoint</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M141.4,-159.876C123.002,-149.232 99.3914,-135.574 80.7766,-124.805\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"82.493,-121.754 72.0844,-119.776 78.9877,-127.813 82.493,-121.754\"/>\n",
       "</g>\n",
       "<!-- prediction1 -->\n",
       "<g id=\"node7\" class=\"node\"><title>prediction1</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"180,-124 102,-124 102,-88 180,-88 180,-124\"/>\n",
       "<text text-anchor=\"middle\" x=\"141\" y=\"-102.3\" font-family=\"Times,serif\" font-size=\"14.00\">prediction1</text>\n",
       "</g>\n",
       "<!-- imagenet&#45;combiner&#45;&gt;prediction1 -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>imagenet&#45;combiner&#45;&gt;prediction1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M163.584,-159.697C160.206,-151.813 156.13,-142.304 152.377,-133.546\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"155.487,-131.917 148.33,-124.104 149.053,-134.675 155.487,-131.917\"/>\n",
       "</g>\n",
       "<!-- prediction2 -->\n",
       "<g id=\"node9\" class=\"node\"><title>prediction2</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"276,-124 198,-124 198,-88 276,-88 276,-124\"/>\n",
       "<text text-anchor=\"middle\" x=\"237\" y=\"-102.3\" font-family=\"Times,serif\" font-size=\"14.00\">prediction2</text>\n",
       "</g>\n",
       "<!-- imagenet&#45;combiner&#45;&gt;prediction2 -->\n",
       "<g id=\"edge7\" class=\"edge\"><title>imagenet&#45;combiner&#45;&gt;prediction2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M187.315,-159.697C195.307,-151.22 205.072,-140.864 213.822,-131.583\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"216.559,-133.781 220.873,-124.104 211.466,-128.979 216.559,-133.781\"/>\n",
       "</g>\n",
       "<!-- prediction1endpoint -->\n",
       "<g id=\"node8\" class=\"node\"><title>prediction1endpoint</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"141\" cy=\"-34\" rx=\"33.5952\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"141\" y=\"-30.3\" font-family=\"Times,serif\" font-size=\"14.00\">GRPC</text>\n",
       "</g>\n",
       "<!-- prediction1&#45;&gt;prediction1endpoint -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>prediction1&#45;&gt;prediction1endpoint</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M141,-87.6966C141,-79.9827 141,-70.7125 141,-62.1124\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"144.5,-62.1043 141,-52.1043 137.5,-62.1044 144.5,-62.1043\"/>\n",
       "</g>\n",
       "<!-- prediction2endpoint -->\n",
       "<g id=\"node10\" class=\"node\"><title>prediction2endpoint</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"237\" cy=\"-34\" rx=\"33.5952\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"237\" y=\"-30.3\" font-family=\"Times,serif\" font-size=\"14.00\">GRPC</text>\n",
       "</g>\n",
       "<!-- prediction2&#45;&gt;prediction2endpoint -->\n",
       "<g id=\"edge6\" class=\"edge\"><title>prediction2&#45;&gt;prediction2endpoint</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M237,-87.6966C237,-79.9827 237,-70.7125 237,-62.1124\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"240.5,-62.1043 237,-52.1043 233.5,-62.1044 240.5,-62.1043\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7fc5004e6198>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_graph(\"seldon_ov_predict_ensemble.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \u001b[34;01m\"apiVersion\"\u001b[39;49;00m: \u001b[33m\"machinelearning.seldon.io/v1alpha2\"\u001b[39;49;00m,\r\n",
      "  \u001b[34;01m\"kind\"\u001b[39;49;00m: \u001b[33m\"SeldonDeployment\"\u001b[39;49;00m,\r\n",
      "  \u001b[34;01m\"metadata\"\u001b[39;49;00m: {\r\n",
      "    \u001b[34;01m\"labels\"\u001b[39;49;00m: {\r\n",
      "      \u001b[34;01m\"app\"\u001b[39;49;00m: \u001b[33m\"seldon\"\u001b[39;49;00m\r\n",
      "    },\r\n",
      "    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"openvino-model\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"namespace\"\u001b[39;49;00m: \u001b[33m\"seldon\"\u001b[39;49;00m\r\n",
      "  },\r\n",
      "    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "\t\u001b[34;01m\"annotations\"\u001b[39;49;00m : {\r\n",
      "\t    \u001b[34;01m\"seldon.io/grpc-read-timeout\"\u001b[39;49;00m:\u001b[33m\"100000\"\u001b[39;49;00m\r\n",
      "\t},\r\n",
      "    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"openvino\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"predictors\"\u001b[39;49;00m: [\r\n",
      "      {\r\n",
      "        \u001b[34;01m\"componentSpecs\"\u001b[39;49;00m: [{\r\n",
      "          \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"containers\"\u001b[39;49;00m: [\r\n",
      "              {\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"imagenet-itransformer\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/openvino-demo-transformer:0.1\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"env\"\u001b[39;49;00m: [\r\n",
      "                  {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"TRACING\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"1\"\u001b[39;49;00m\r\n",
      "                  },\r\n",
      "                  {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"JAEGER_AGENT_HOST\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"jaeger-agent\"\u001b[39;49;00m\r\n",
      "                  },\r\n",
      "                  {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"DTYPE\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"float32\"\u001b[39;49;00m\r\n",
      "                  }\r\n",
      "                ]\r\n",
      "              },\r\n",
      "              {\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"imagenet-otransformer\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/openvino-demo-transformer:0.1\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"env\"\u001b[39;49;00m: [\r\n",
      "                  {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"TRACING\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"1\"\u001b[39;49;00m\r\n",
      "                  },\r\n",
      "                  {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"JAEGER_AGENT_HOST\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"jaeger-agent\"\u001b[39;49;00m\r\n",
      "                  }\r\n",
      "                ]\r\n",
      "              },\r\n",
      "              {\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"imagenet-combiner\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/openvino-demo-combiner:0.1\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"env\"\u001b[39;49;00m: [\r\n",
      "                  {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"TRACING\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"1\"\u001b[39;49;00m\r\n",
      "                  },\r\n",
      "                  {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"JAEGER_AGENT_HOST\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"jaeger-agent\"\u001b[39;49;00m\r\n",
      "                  }\r\n",
      "                ]\r\n",
      "              },\r\n",
      "              {\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"prediction1\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/openvino-demo-prediction:0.1\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"env\"\u001b[39;49;00m: [\r\n",
      "                  {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"XML_PATH\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"gs://intelai_public_models/densenet_169/1/densenet_169_i8.xml\"\u001b[39;49;00m\r\n",
      "                  },\r\n",
      "                  {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"BIN_PATH\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"gs://intelai_public_models/densenet_169/1/densenet_169_i8.bin\"\u001b[39;49;00m\r\n",
      "                  },\r\n",
      "                  {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"http_proxy\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"\"\u001b[39;49;00m\r\n",
      "                  },\r\n",
      "                  {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"https_proxy\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"\"\u001b[39;49;00m\r\n",
      "                  },\r\n",
      "                  {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"TRACING\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"1\"\u001b[39;49;00m\r\n",
      "                  },\r\n",
      "                  {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"JAEGER_AGENT_HOST\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"jaeger-agent\"\u001b[39;49;00m\r\n",
      "                  }\r\n",
      "                ]\r\n",
      "              },\r\n",
      "              {\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"prediction2\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/openvino-demo-prediction:0.1\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"env\"\u001b[39;49;00m: [\r\n",
      "                   {\r\n",
      "                     \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"XML_PATH\"\u001b[39;49;00m,\r\n",
      "                     \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"gs://intelai_public_models/resnet_50_i8/1/resnet_50_i8.xml\"\u001b[39;49;00m\r\n",
      "                   },\r\n",
      "                   {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"BIN_PATH\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"gs://intelai_public_models/resnet_50_i8/1/resnet_50_i8.bin\"\u001b[39;49;00m\r\n",
      "                   },\r\n",
      "                   {\r\n",
      "                     \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"http_proxy\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"\"\u001b[39;49;00m\r\n",
      "                   },\r\n",
      "                   {\r\n",
      "                     \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"https_proxy\"\u001b[39;49;00m,\r\n",
      "                     \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"\"\u001b[39;49;00m\r\n",
      "                   },\r\n",
      "                   {\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"TRACING\"\u001b[39;49;00m,\r\n",
      "                     \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"1\"\u001b[39;49;00m\r\n",
      "                   },\r\n",
      "                   {\r\n",
      "                     \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"JAEGER_AGENT_HOST\"\u001b[39;49;00m,\r\n",
      "                     \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"jaeger-agent\"\u001b[39;49;00m\r\n",
      "                   }\r\n",
      "                  ]\r\n",
      "              }\r\n",
      "            ],\r\n",
      "            \u001b[34;01m\"terminationGracePeriodSeconds\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m\r\n",
      "          }\r\n",
      "        }],\r\n",
      "        \u001b[34;01m\"graph\"\u001b[39;49;00m: {\r\n",
      "          \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"imagenet-otransformer\"\u001b[39;49;00m,\r\n",
      "          \u001b[34;01m\"endpoint\"\u001b[39;49;00m: { \u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"GRPC\"\u001b[39;49;00m },\r\n",
      "          \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"OUTPUT_TRANSFORMER\"\u001b[39;49;00m,\r\n",
      "          \u001b[34;01m\"children\"\u001b[39;49;00m: [\r\n",
      "            {\r\n",
      "\r\n",
      "              \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"imagenet-itransformer\"\u001b[39;49;00m,\r\n",
      "              \u001b[34;01m\"endpoint\"\u001b[39;49;00m: { \u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"GRPC\"\u001b[39;49;00m },\r\n",
      "              \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"TRANSFORMER\"\u001b[39;49;00m,\r\n",
      "              \u001b[34;01m\"children\"\u001b[39;49;00m: [\r\n",
      "                {\r\n",
      "                  \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"imagenet-combiner\"\u001b[39;49;00m,\r\n",
      "                  \u001b[34;01m\"endpoint\"\u001b[39;49;00m: { \u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"GRPC\"\u001b[39;49;00m },\r\n",
      "                  \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"COMBINER\"\u001b[39;49;00m,\r\n",
      "                  \u001b[34;01m\"children\"\u001b[39;49;00m: [\r\n",
      "                    {\r\n",
      "                      \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"prediction1\"\u001b[39;49;00m,\r\n",
      "                      \u001b[34;01m\"endpoint\"\u001b[39;49;00m: { \u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"GRPC\"\u001b[39;49;00m },\r\n",
      "                      \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"MODEL\"\u001b[39;49;00m,\r\n",
      "                      \u001b[34;01m\"children\"\u001b[39;49;00m: []\r\n",
      "                    },\r\n",
      "                    {\r\n",
      "                      \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"prediction2\"\u001b[39;49;00m,\r\n",
      "                      \u001b[34;01m\"endpoint\"\u001b[39;49;00m: { \u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"GRPC\"\u001b[39;49;00m },\r\n",
      "                      \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"MODEL\"\u001b[39;49;00m,\r\n",
      "                      \u001b[34;01m\"children\"\u001b[39;49;00m: []\r\n",
      "                    }\r\n",
      "                  ]\r\n",
      "                }\r\n",
      "              ]\r\n",
      "            }\r\n",
      "          ]\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"openvino\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"replicas\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"svcOrchSpec\"\u001b[39;49;00m : {\r\n",
      "          \u001b[34;01m\"env\"\u001b[39;49;00m: [\r\n",
      "            {\r\n",
      "              \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"TRACING\"\u001b[39;49;00m,\r\n",
      "              \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"1\"\u001b[39;49;00m\r\n",
      "            },\r\n",
      "            {\r\n",
      "              \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"JAEGER_AGENT_HOST\"\u001b[39;49;00m,\r\n",
      "              \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"jaeger-agent\"\u001b[39;49;00m\r\n",
      "            },\r\n",
      "            {\r\n",
      "              \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"JAEGER_AGENT_PORT\"\u001b[39;49;00m,\r\n",
      "              \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"5775\"\u001b[39;49;00m\r\n",
      "            },\r\n",
      "            {\r\n",
      "              \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"JAEGER_SAMPLER_TYPE\"\u001b[39;49;00m,\r\n",
      "              \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"const\"\u001b[39;49;00m\r\n",
      "            },\r\n",
      "            {\r\n",
      "              \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"JAEGER_SAMPLER_PARAM\"\u001b[39;49;00m,\r\n",
      "              \u001b[34;01m\"value\"\u001b[39;49;00m: \u001b[33m\"1\"\u001b[39;49;00m\r\n",
      "            }\r\n",
      "          ]\r\n",
      "        }\r\n",
      "      }\r\n",
      "    ]\r\n",
      "  }\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize seldon_ov_predict_ensemble.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/openvino-model created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f seldon_ov_predict_ensemble.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Executing the pipeline\n",
    "\n",
    "### Connectivity with the seldon pipeline\n",
    "\n",
    "You may connect to the seldon ambassador endpoint using on of the following options:\n",
    "- Establish a tunnel over http via `kubectl port-forward` command.\n",
    "\n",
    "Expose ambassador API endpoint outside of the Kubernetes cluster or connect to it via `kubectl port-forward`.\n",
    "\n",
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8080:8080\n",
    "```\n",
    "- Expose the service `seldon-core-ambassador` using a `LoadBalancer` or `NodePort` type.\n",
    "\n",
    "```kubectl edit service seldon-core-ambassador```\n",
    "\n",
    "Check the assigned External IP address with:\n",
    "\n",
    "```kubectl get service seldon-core-ambassador```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the exemplary grpc client\n",
    "\n",
    "Install client dependencies: seldon-core and grpcio packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mYou are using pip version 19.0.3, however version 19.1 is available.\r\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!pip install -q seldon-core grpcio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "meta {\n",
      "  puid: \"meabbkemsugfo0vc84j7vc19us\"\n",
      "  routing {\n",
      "    key: \"imagenet-combiner\"\n",
      "    value: -1\n",
      "  }\n",
      "  routing {\n",
      "    key: \"imagenet-itransformer\"\n",
      "    value: -1\n",
      "  }\n",
      "  routing {\n",
      "    key: \"imagenet-otransformer\"\n",
      "    value: -1\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"imagenet-combiner\"\n",
      "    value: \"seldonio/openvino-demo-combiner:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"imagenet-itransformer\"\n",
      "    value: \"seldonio/openvino-demo-transformer:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"imagenet-otransformer\"\n",
      "    value: \"seldonio/openvino-demo-transformer:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"prediction1\"\n",
      "    value: \"seldonio/openvino-demo-prediction:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"prediction2\"\n",
      "    value: \"seldonio/openvino-demo-prediction:0.1\"\n",
      "  }\n",
      "}\n",
      "strData: \"Eskimo dog, husky\"\n",
      "\n",
      "Duration 8459.775 ms\n",
      "meta {\n",
      "  puid: \"3l9jq1tm0c5lu00gp476hca9bd\"\n",
      "  routing {\n",
      "    key: \"imagenet-combiner\"\n",
      "    value: -1\n",
      "  }\n",
      "  routing {\n",
      "    key: \"imagenet-itransformer\"\n",
      "    value: -1\n",
      "  }\n",
      "  routing {\n",
      "    key: \"imagenet-otransformer\"\n",
      "    value: -1\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"imagenet-combiner\"\n",
      "    value: \"seldonio/openvino-demo-combiner:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"imagenet-itransformer\"\n",
      "    value: \"seldonio/openvino-demo-transformer:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"imagenet-otransformer\"\n",
      "    value: \"seldonio/openvino-demo-transformer:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"prediction1\"\n",
      "    value: \"seldonio/openvino-demo-prediction:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"prediction2\"\n",
      "    value: \"seldonio/openvino-demo-prediction:0.1\"\n",
      "  }\n",
      "}\n",
      "strData: \"zebra\"\n",
      "\n",
      "Duration 7514.488 ms\n",
      "meta {\n",
      "  puid: \"467pdas9vi2mb0t0hrvjajud4s\"\n",
      "  routing {\n",
      "    key: \"imagenet-combiner\"\n",
      "    value: -1\n",
      "  }\n",
      "  routing {\n",
      "    key: \"imagenet-itransformer\"\n",
      "    value: -1\n",
      "  }\n",
      "  routing {\n",
      "    key: \"imagenet-otransformer\"\n",
      "    value: -1\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"imagenet-combiner\"\n",
      "    value: \"seldonio/openvino-demo-combiner:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"imagenet-itransformer\"\n",
      "    value: \"seldonio/openvino-demo-transformer:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"imagenet-otransformer\"\n",
      "    value: \"seldonio/openvino-demo-transformer:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"prediction1\"\n",
      "    value: \"seldonio/openvino-demo-prediction:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"prediction2\"\n",
      "    value: \"seldonio/openvino-demo-prediction:0.1\"\n",
      "  }\n",
      "}\n",
      "strData: \"pelican\"\n",
      "\n",
      "Duration 8559.143 ms\n",
      "average duration: 8177.802 ms\n",
      "average accuracy: 100.0\n"
     ]
    }
   ],
   "source": [
    "!python seldon_grpc_client.py --debug"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more extensive test see the client help.\n",
    "\n",
    "You can change the default test-input file including labeled list of images to calculate accuracy based on complete imagenet dataset. Follow the format from file `input_images.txt` - path to the image and imagenet class in every line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: seldon_grpc_client.py [-h] [--repeats REPEATS] [--debug]\r\n",
      "                             [--test-input TEST_INPUT]\r\n",
      "\r\n",
      "optional arguments:\r\n",
      "  -h, --help            show this help message and exit\r\n",
      "  --repeats REPEATS\r\n",
      "  --debug\r\n",
      "  --test-input TEST_INPUT\r\n"
     ]
    }
   ],
   "source": [
    "!python seldon_grpc_client.py --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examining the logs\n",
    "\n",
    "You can use Seldon containers logs to get additional details about the execution:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-02-12 10:32:38,646 - Prediction - DEBUG - Processing time: 27.29 ms\r\n",
      "2019-02-12 10:32:38,646 - Prediction:predict:103 - DEBUG:  Processing time: 27.29 ms\r\n",
      "2019-02-12 10:32:38,699 - Prediction - DEBUG - Processing time: 26.76 ms\r\n",
      "2019-02-12 10:32:38,699 - Prediction:predict:103 - DEBUG:  Processing time: 26.76 ms\r\n",
      "2019-02-12 10:37:59,123 - Prediction - DEBUG - Processing time: 27.28 ms\r\n",
      "2019-02-12 10:37:59,123 - Prediction:predict:103 - DEBUG:  Processing time: 27.28 ms\r\n",
      "2019-02-12 10:37:59,174 - Prediction - DEBUG - Processing time: 26.20 ms\r\n",
      "2019-02-12 10:37:59,174 - Prediction:predict:103 - DEBUG:  Processing time: 26.20 ms\r\n",
      "2019-02-12 10:37:59,228 - Prediction - DEBUG - Processing time: 27.33 ms\r\n",
      "2019-02-12 10:37:59,228 - Prediction:predict:103 - DEBUG:  Processing time: 27.33 ms\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl logs $(kubectl get pods -l seldon-app=openvino -o jsonpath='{.items[0].metadata.name}') prediction1 --tail=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-02-12 10:32:38,630 - Prediction - DEBUG - Processing time: 9.86 ms\r\n",
      "2019-02-12 10:32:38,630 - Prediction:predict:103 - DEBUG:  Processing time: 9.86 ms\r\n",
      "2019-02-12 10:32:38,681 - Prediction - DEBUG - Processing time: 9.32 ms\r\n",
      "2019-02-12 10:32:38,681 - Prediction:predict:103 - DEBUG:  Processing time: 9.32 ms\r\n",
      "2019-02-12 10:37:59,111 - Prediction - DEBUG - Processing time: 15.16 ms\r\n",
      "2019-02-12 10:37:59,111 - Prediction:predict:103 - DEBUG:  Processing time: 15.16 ms\r\n",
      "2019-02-12 10:37:59,158 - Prediction - DEBUG - Processing time: 9.16 ms\r\n",
      "2019-02-12 10:37:59,158 - Prediction:predict:103 - DEBUG:  Processing time: 9.16 ms\r\n",
      "2019-02-12 10:37:59,211 - Prediction - DEBUG - Processing time: 9.62 ms\r\n",
      "2019-02-12 10:37:59,211 - Prediction:predict:103 - DEBUG:  Processing time: 9.62 ms\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl logs $(kubectl get pods -l seldon-app=openvino -o jsonpath='{.items[0].metadata.name}') prediction2 --tail=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-02-12 10:37:59,086 - ImageNetTransformer:transform_input_grpc:43 - INFO:  jpeg preprocessing: 1.464 ms\r\n",
      "2019-02-12 10:37:59,089 - ImageNetTransformer:transform_input_grpc:50 - INFO:  Total transformation: 4.042 ms\r\n",
      "2019-02-12 10:37:59,137 - ImageNetTransformer:transform_input_grpc:33 - INFO:  Transform called\r\n",
      "2019-02-12 10:37:59,139 - ImageNetTransformer:transform_input_grpc:40 - INFO:  Shape: (1, 3, 224, 224); Dtype: float32; Min: 0.0; Max: 255.0\r\n",
      "2019-02-12 10:37:59,140 - ImageNetTransformer:transform_input_grpc:43 - INFO:  jpeg preprocessing: 2.222 ms\r\n",
      "2019-02-12 10:37:59,142 - ImageNetTransformer:transform_input_grpc:50 - INFO:  Total transformation: 4.92 ms\r\n",
      "2019-02-12 10:37:59,188 - ImageNetTransformer:transform_input_grpc:33 - INFO:  Transform called\r\n",
      "2019-02-12 10:37:59,191 - ImageNetTransformer:transform_input_grpc:40 - INFO:  Shape: (1, 3, 224, 224); Dtype: float32; Min: 0.0; Max: 255.0\r\n",
      "2019-02-12 10:37:59,191 - ImageNetTransformer:transform_input_grpc:43 - INFO:  jpeg preprocessing: 2.3249999999999997 ms\r\n",
      "2019-02-12 10:37:59,194 - ImageNetTransformer:transform_input_grpc:50 - INFO:  Total transformation: 5.760999999999999 ms\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl logs $(kubectl get pods -l seldon-app=openvino -o jsonpath='{.items[0].metadata.name}') imagenet-itransformer --tail=10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performance consideration\n",
    "\n",
    "In production environment with a shared workloads, you might consider contraining the CPU resources for individual pipeline components. You might restrict the assigned capacity using [Kubernetes capabilities](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/). This configuration can be added to seldon pipeline definition.\n",
    "\n",
    "Another option for tuning the resource allocation is adding environment variable `OMP_NUM_THREADS`. It can indicate how many threads will be used by OpenVINO execution engine and how many CPU cores can be consumed. The recommeded value is equal to the number of allocated CPU physical cores.\n",
    "\n",
    "In the tests using GKE service in Google Cloud on nodes with 32 SkyLake vCPU assigned, the following configuration was set on prediction components. It achieved the optimal latency and throughput:\n",
    "```\n",
    "\"resources\": {\n",
    "  \"requests\": {\n",
    "     \"cpu\": \"1\"\n",
    "  },\n",
    "  \"limits\": {\n",
    "     \"cpu\": \"32\"\n",
    "  }\n",
    "}\n",
    "\n",
    "\"env\": [\n",
    "  {\n",
    "    \"name\": \"KMP_AFFINITY\",\n",
    "    \"value\": \"granularity=fine,verbose,compact,1,0\"\n",
    "  },\n",
    "  {\n",
    "    \"name\": \"KMP_BLOCKTIME\",\n",
    "    \"value\": \"1\"\n",
    "  },\n",
    "  {\n",
    "    \"name\": \"OMP_NUM_THREADS\",\n",
    "    \"value\": \"8\"\n",
    "  }\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
